Wednesday, April 9, 2008
We're Back...And Here's Our Tale of Woe
Posted by Jason Dunn in "Thoughts Media Status Updates" @ 08:51 AM
After what is, I believe, the longest down-time in the history of these sites, we're back and live. Such a long window of down-time deserves an explanation more detailed than a haiku (did anyone notice the first version of the haiku wasn't actually a proper haiku? And this is from the guy who just got back from Japan!), but this explanation has to reach back a few years to make complete sense.
We've been running on the same main machine for web and database serving for several years - a dual Xeon box from ev1 (we also had a second box for email and image serving). Long story short(er), it was largely unmanaged - I got help for specific problems when I needed it, but by and large it was completely left alone for almost four years. Yeah, that's very bad news from a security standpoint. I'm NOT a server guy, so I think I buried my head in the sand and hoped for the best. Not a good plan, right? Part of the problem was that the OS was old and what we really needed was to move to a new box with a new OS - but I didn't want to do that until all the sites were on vBulletin and I knew how much hardware I needed (phpBB is much more resource intensive than vBulletin is). Getting from phpBB to vBulletin took, literally, almost two years of development work - we had to develop a new CMS from scratch, something flexible and powerful enough to be used for years to come.
Somehow, miraculously, over the years we never had major problems on our main server (likely because it had no control panel software to hack) - until December of last year. A few days after my dear friend Crystal passed away, and during the weekend when I was preparing for her funeral (I was doing some video/photos stuff for it), our secondary server (email/images) got hacked via Cpanel. So while grieving her loss I was also dealing with having huge email problems and worrying about losing data. Over the next couple of days, Jorj was moving that server to a virtual machine on our main server. Incredibly, as if the universe itself was punishing me, the morning of the funeral day our other server got hacked and our install of phpBB was breached several times. No data was lost, but we had spammers and virus/spyware distributors trying to hijack our forums. We fought them off, and closed the holes as best we could. I can't explain how grateful I am to Jorj and Janak for their help - Jorj in particular spent a great deal of time working on the server in between celebrating Christmas with his family (and this is all volunteered time). His selflessness was/is awe-inspiring. Read more...
We've been running on the same main machine for web and database serving for several years - a dual Xeon box from ev1 (we also had a second box for email and image serving). Long story short(er), it was largely unmanaged - I got help for specific problems when I needed it, but by and large it was completely left alone for almost four years. Yeah, that's very bad news from a security standpoint. I'm NOT a server guy, so I think I buried my head in the sand and hoped for the best. Not a good plan, right? Part of the problem was that the OS was old and what we really needed was to move to a new box with a new OS - but I didn't want to do that until all the sites were on vBulletin and I knew how much hardware I needed (phpBB is much more resource intensive than vBulletin is). Getting from phpBB to vBulletin took, literally, almost two years of development work - we had to develop a new CMS from scratch, something flexible and powerful enough to be used for years to come.
Somehow, miraculously, over the years we never had major problems on our main server (likely because it had no control panel software to hack) - until December of last year. A few days after my dear friend Crystal passed away, and during the weekend when I was preparing for her funeral (I was doing some video/photos stuff for it), our secondary server (email/images) got hacked via Cpanel. So while grieving her loss I was also dealing with having huge email problems and worrying about losing data. Over the next couple of days, Jorj was moving that server to a virtual machine on our main server. Incredibly, as if the universe itself was punishing me, the morning of the funeral day our other server got hacked and our install of phpBB was breached several times. No data was lost, but we had spammers and virus/spyware distributors trying to hijack our forums. We fought them off, and closed the holes as best we could. I can't explain how grateful I am to Jorj and Janak for their help - Jorj in particular spent a great deal of time working on the server in between celebrating Christmas with his family (and this is all volunteered time). His selflessness was/is awe-inspiring. Read more...
